A Goal-Driven Cognitive Model Of The Systems Evaluation Process

Researchers have tried to build valid and reliable instruments for measuring users' assessments of IS. However, focusing on instruments is a somewhat ineffective way of dealing with problems associated with the evaluation process itself. This paper presents a model of the process by which individuals evaluate an IS. The model emphasizes the importance of raters' goals. To form an evaluation, raters seek information about an IS and compare it to their normative beliefs. The information they seek is influenced by their task support goals. Evaluations can be adjusted to serve raters' social adjustment, value expressive, and protective goals. Empirical evidence supporting the model is discussed, along with the model's implications for system developers. Some suggestions for future research are made. 1. INTRODUCI'ION information satisfaction (UIS) (e.g., Bailey and Pearson 1983; Doll and Torkzadeh 1988; Ives, Olson and Baroudi Users' evaluations of information systems (IS) are impor1983; Jenkins and Ricketts 1985). Some researchers have tant for two main reasons. First, user evaluations provide identified limitations of UIS instruments. For instance, information to help guide development. In prototyping Galletta and Lederer (1989) note problems with scale units (Naumann and Jenkins 1982), user evaluations are a and origins, item heterogeneity, and reliability. Ryan and central part of the project control mechanism. A developBock (1990) have suggested that the ambiguity of items in er gives a prototype to a user for evaluation and changes UIS instruments complicates the interpretation of rater's the prototype based on the user's suggestions for improveresponses. However, some of these problems are not ments. If the user's ideas about the system are incorrect, characteristics of the instruments per se, but of the way then development could proceed in the wrong direction. they are used. For instance, some users have little contact with IS developers and cannot evaluate them accurately, no More recent development methodologies also emphasize matter what the instrument. the importance of user evaluations. For example, Martin's (1991) Rapid Application Development methodology The current literature on users' evaluation resembles the involves four different user teams who evaluate everything literature on employee performance appraisal as it was a from requirements specifications to general design to the decade ago. Traditionally, researchers focused their efforts final system. Martin claims that users should actively on appraisal instruments, trying to reduce bias and increase participate in every phase of IS development. reliability (Landy and Farr 1980). However, Ilgen and Favero (1985) noted that focusing on scale construction is The second reason that user evaluations are important is a weak and indirect way of dealing with what are really that the evaluations guide the behavior of the users problems in human judgment. Landy and Farr (1980) themselves. For example, Davis, Bagozzi and Warshaw recommended a shift away from research on instruments (1989) found that individuals' intentions to use an IS are to research aimed at understanding how raters evaluate influenced by their evaluations of a system's usefulness and employees. In response, researchers have developed ease of use. If their evaluations are not accurate, users models of the evaluation process (e.g., DeNisi, Caferty and could act incorrectly. On the one hand, they might waste Meglino 1984). Later work has reflected a concern for resources on a system that initially seemed valuable but process issues. For instance, Pulakos (1986) developed turned out not to be effective. On the other hand, users successful rater training programs based on the implicacould miss an opportunity to improve performance by tions of process models. choosing not to use a system that has value. Given the importance of users' evaluations of IS, similar Researchers have designed instruments to quantify users' efforts by MIS researchers are warranted. This paper evaluations. Much of this work has focused on user introduces a model of the process by which an individual